{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "Ltj0f-flNAi9"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "##### Copyright 2023 Google LLC. SPDX-License-Identifier: Apache-2.0"
      ],
      "metadata": {
        "id": "Ltj0f-flNAi9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Copyright 2023 Google LLC. SPDX-License-Identifier: Apache-2.0\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at\n",
        "\n",
        "https://www.apache.org/licenses/LICENSE-2.0\n",
        "\n",
        "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
      ],
      "metadata": {
        "id": "I8NlVpAzNB2u"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **LLMs as General Pattern Machines:** ARC Benchmark\n",
        "\n",
        "We observe that pretrained large language models (LLMs) are capable of autoregressively completing complex token sequences -- from arbitrary ones procedurally generated by probabilistic context-free grammars (PCFG), to more rich spatial patterns found in the Abstract Reasoning Corpus (ARC), a general AI benchmark, prompted in the style of ASCII art. Surprisingly, pattern completion proficiency can be partially retained even when the sequences are expressed using tokens randomly sampled from the vocabulary. These results suggest that without any additional training, LLMs can serve as general sequence modelers, driven by in-context learning. In this work, we investigate how these zero-shot capabilities may be applied to problems in robotics -- from extrapolating sequences of numbers that represent states over time to complete simple motions, to least-to-most prompting of reward-conditioned trajectories that can discover and represent closed-loop policies (e.g., a stabilizing controller for CartPole). While difficult to deploy today for real systems due to latency, context size limitations, and compute costs, the approach of using LLMs to drive low-level control may provide an exciting glimpse into how the patterns among words could be transferred to actions.\n",
        "\n",
        "This colab runs GPT-3 on the ARC benchmark with consistent tokenization (described more in Sec. 4 of the main paper).\n",
        "\n",
        "### **Quick Start:**\n",
        "\n",
        "**Step 1.** Register for an [OpenAI API key](https://openai.com/blog/openai-api/) to use GPT-3 (there's a free trial) and enter it below\n",
        "\n",
        "**Step 2.** Menu > Runtime > Run all"
      ],
      "metadata": {
        "id": "zqTADtDB6zyA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "openai_api_key = \"your-api-key-here\""
      ],
      "metadata": {
        "id": "wwJDOJSz71lk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Setup**\n",
        "\n",
        "This does a few things:\n",
        "* Installs Python packages and sets OpenAI API key.\n",
        "* Downloads the Abstract Reasoning Corpus (ARC) benchmark.\n",
        "\n",
        "**Note:** only needs a CPU (public) runtime."
      ],
      "metadata": {
        "id": "kbWMlIj7XxX8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install openai transformers\n",
        "\n",
        "import json\n",
        "import os\n",
        "import time\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import openai\n",
        "import pickle\n",
        "from transformers import GPT2Tokenizer\n",
        "# import tiktoken  # Faster than GPT2Tokenizer.\n",
        "\n",
        "openai.api_key = openai_api_key\n",
        "\n",
        "if not os.path.exists(\"ARC\"):\n",
        "  !git clone https://github.com/fchollet/ARC"
      ],
      "metadata": {
        "id": "uI4hX8y5XzeH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **API:** Large Language Models\n",
        "\n",
        "Define helper functions to call large language models and the tokenizer.\n",
        "\n",
        "**Note:** this can get expensive."
      ],
      "metadata": {
        "id": "2AoMDZ-GZxRP"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-waqt2fUb9ex",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f2d6d52e-5502-45e7-91cb-0855fde30c60"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[\"\\n\\nHello World! It's great to be here.\"]"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ],
      "source": [
        "model = \"text-davinci-003\"\n",
        "token_limit = 4096\n",
        "\n",
        "def LLM(prompt, stop=None, max_tokens=256, temperature=0):\n",
        "  responses = openai.Completion.create(engine=model, prompt=prompt, max_tokens=max_tokens, temperature=temperature, stop=stop)\n",
        "  text = [response['text'] for response in responses['choices']]\n",
        "  return text\n",
        "\n",
        "tokenizer = GPT2Tokenizer.from_pretrained(\"gpt2\")\n",
        "\n",
        "LLM(\"hello world!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Alphabet:** Token Set\n",
        "\n",
        "Build a fixed token set by random sampling from the LLM's token vocabulary."
      ],
      "metadata": {
        "id": "vMPLptkxatkC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "item_delim = tokenizer.encode(\",\")\n",
        "row_delim = tokenizer.encode(\"\\n\")\n",
        "sample_delim = tokenizer.encode(\"---\\n\")\n",
        "\n",
        "# Handpicked: comma-separated number matrices.\n",
        "alphabet = [tokenizer.encode(\" \" + str(a))[0] for a in range(10)]\n",
        "value_to_token = lambda x: {i:a for i, a in enumerate(alphabet)}[x]\n",
        "print(\"Token Set:\", {i:value_to_token(i) for i in np.arange(10)})\n",
        "\n",
        "# Random sampled tokens.\n",
        "# for seed_offset in range(20):\n",
        "# seed_offset = 0\n",
        "# np.random.seed(42 + seed_offset)\n",
        "# alphabet = [int(i) for i in np.random.randint(tokenizer.vocab_size, size=10)]\n",
        "# value_to_token = lambda x: {i:a for i, a in enumerate(alphabet)}[x]"
      ],
      "metadata": {
        "id": "tRIQ3AZMUmpK",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9e07e590-0986-4320-f164-0268609a9cf7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Token Set: {0: 657, 1: 352, 2: 362, 3: 513, 4: 604, 5: 642, 6: 718, 7: 767, 8: 807, 9: 860}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Load:** ARC Benchmark\n",
        "\n",
        "Load tasks from the ARC benchmark."
      ],
      "metadata": {
        "id": "SD_-rknea0WU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def state_to_tokens(state, value_to_token_fn):\n",
        "  tokens = []\n",
        "  for row in state:\n",
        "    for i, value in enumerate(row):\n",
        "      tokens +=[value_to_token_fn(value)]\n",
        "      if i < len(row) - 1:\n",
        "        tokens += item_delim\n",
        "    tokens += row_delim\n",
        "  return tokens\n",
        "\n",
        "\n",
        "def task_json_to_tokens(task_json, value_to_token_fn):\n",
        "\n",
        "  # Training examples.\n",
        "  train_samples = []\n",
        "  for sample in task_json[\"train\"]:\n",
        "    tokens = []\n",
        "    tokens += tokenizer.encode(\"input:\\n\")\n",
        "    tokens += state_to_tokens(sample[\"input\"], value_to_token_fn)\n",
        "    tokens += tokenizer.encode(\"output:\\n\")\n",
        "    tokens += state_to_tokens(sample[\"output\"], value_to_token_fn)\n",
        "    tokens += sample_delim\n",
        "    train_samples.append(tokens)\n",
        "\n",
        "  # Testing examples.\n",
        "  test_inputs = []\n",
        "  test_outputs = []\n",
        "  for sample in task_json[\"test\"]:\n",
        "    inputs, outputs = [], []\n",
        "    inputs += tokenizer.encode(\"input:\\n\")\n",
        "    inputs += state_to_tokens(sample[\"input\"], value_to_token_fn)\n",
        "    inputs += tokenizer.encode(\"output:\\n\")\n",
        "    test_inputs.append(inputs)\n",
        "    outputs += state_to_tokens(sample[\"output\"], value_to_token_fn)\n",
        "    test_outputs.append(outputs)\n",
        "  return train_samples, test_inputs, test_outputs"
      ],
      "metadata": {
        "id": "5rvACw0XFZWY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tasks_jsons = []\n",
        "tasks_names = []\n",
        "tasks_len = []\n",
        "task_dir = \"ARC/data/training\"\n",
        "for task_file in sorted(os.listdir(task_dir)):\n",
        "  with open(os.path.join(task_dir, task_file)) as fid:\n",
        "    task_json = json.load(fid)\n",
        "  tasks_jsons.append(task_json)\n",
        "  tasks_names.append(task_file)\n",
        "  tokens, _, _ = task_json_to_tokens(task_json, value_to_token)\n",
        "  tasks_len.append(np.sum([len(sample) for sample in tokens]))\n",
        "\n",
        "task_dir = \"ARC/data/evaluation\"\n",
        "for task_file in sorted(os.listdir(task_dir)):\n",
        "  with open(os.path.join(task_dir, task_file)) as fid:\n",
        "    task_json = json.load(fid)\n",
        "  tasks_jsons.append(task_json)\n",
        "  tasks_names.append(task_file)\n",
        "  tokens, _, _ = task_json_to_tokens(task_json, value_to_token)\n",
        "  tasks_len.append(np.sum([len(sample) for sample in tokens]))\n",
        "\n",
        "sorted_task_ids = np.argsort(tasks_len)\n",
        "\n",
        "print(\"Total number of tasks:\", len(sorted_task_ids))"
      ],
      "metadata": {
        "id": "zZY7OSoHbIRk",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2fda0d4e-1014-4888-dd2a-d76fb4942ff2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total number of tasks: 800\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Example:** ARC Problem\n",
        "\n",
        "Show the LLM prompt for an ARC problem and visualize the grids used as inputs and outputs."
      ],
      "metadata": {
        "id": "2wBok5T59kDT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "colors = [(0, 0, 0),\n",
        "          (0, 116, 217),\n",
        "          (255, 65, 54),\n",
        "          (46, 204, 6),\n",
        "          (255, 220, 0),\n",
        "          (170, 170, 170),\n",
        "          (240, 18, 190),\n",
        "          (255, 133, 27),\n",
        "          (127, 219, 255),\n",
        "          (135, 12, 37)]\n",
        "\n",
        "def grid_to_img(grid):\n",
        "  grid = np.int32(grid)\n",
        "  scale = 10\n",
        "  img = np.zeros((grid.shape[0] * scale + 1, grid.shape[1] * scale + 1, 3), dtype=np.uint8)\n",
        "  for r in range(grid.shape[0]):\n",
        "    for c in range(grid.shape[1]):\n",
        "      img[r*scale+1:(r+1)*scale, c*scale+1:(c+1)*scale, :] = colors[grid[r, c]]\n",
        "  new_img = img.copy()\n",
        "  new_img[0::10, :, :] = np.uint8(np.round((0.7 * np.float32(img[0::10, :, :]) + 0.3 * 255)))\n",
        "  new_img[:, 0::10, :] = np.uint8(np.round((0.7 * np.float32(img[:, 0::10, :]) + 0.3 * 255)))\n",
        "  return new_img"
      ],
      "metadata": {
        "id": "qCl4heCw88MG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "example_json = tasks_jsons[sorted_task_ids[0]]\n",
        "\n",
        "context = []\n",
        "train_xy, test_x, test_y = task_json_to_tokens(example_json, value_to_token)\n",
        "for sample in train_xy:\n",
        "  context += sample\n",
        "context += test_x[0]\n",
        "\n",
        "print(\"PROMPT:\")\n",
        "print(tokenizer.decode(context, skip_special_tokens=True))\n",
        "print(\"SOLUTION:\")\n",
        "print(tokenizer.decode(test_y[0], skip_special_tokens=True))\n",
        "\n",
        "# Show problem.\n",
        "print(\"TRAIN:\")\n",
        "for i, ex in enumerate(example_json[\"train\"]):\n",
        "  in_img = grid_to_img(ex[\"input\"])\n",
        "  out_img = grid_to_img(ex[\"output\"])\n",
        "  plt.subplot(1, 2, 1); plt.imshow(grid_to_img(ex[\"input\"]))\n",
        "  plt.subplot(1, 2, 2); plt.imshow(grid_to_img(ex[\"output\"]))\n",
        "  plt.show()\n",
        "print(\"TEST:\")\n",
        "for i, ex in enumerate(example_json[\"test\"]):\n",
        "  in_img = grid_to_img(ex[\"input\"])\n",
        "  out_img = grid_to_img(ex[\"output\"])\n",
        "  plt.subplot(1, 2, 1); plt.imshow(grid_to_img(ex[\"input\"]))\n",
        "  plt.subplot(1, 2, 2); plt.imshow(grid_to_img(ex[\"output\"]))\n",
        "  plt.show()"
      ],
      "metadata": {
        "id": "orLb7781byY3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "a5e6948a-ccf2-4dc6-bf03-d7902c07e85a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "PROMPT:\n",
            "input:\n",
            " 3, 3, 8\n",
            " 3, 7, 0\n",
            " 5, 0, 0\n",
            "output:\n",
            " 0, 0, 5\n",
            " 0, 7, 3\n",
            " 8, 3, 3\n",
            "---\n",
            "input:\n",
            " 5, 5, 2\n",
            " 1, 0, 0\n",
            " 0, 0, 0\n",
            "output:\n",
            " 0, 0, 0\n",
            " 0, 0, 1\n",
            " 2, 5, 5\n",
            "---\n",
            "input:\n",
            " 6, 3, 5\n",
            " 6, 8, 0\n",
            " 4, 0, 0\n",
            "output:\n",
            "\n",
            "SOLUTION:\n",
            " 0, 0, 4\n",
            " 0, 8, 6\n",
            " 5, 3, 6\n",
            "\n",
            "TRAIN:\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiQAAAEPCAYAAABycN8YAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAZkklEQVR4nO3df0xV9/3H8RcoXK3CpfiDKxEcrU7bWuk3TOmNnbGTiS4abTGZ3RJ1a2pkYGLN0knTVu2W3EaT1bajNNkyqcmsW5ehkaZqi4pphy4yCbVOUg2bNOVia8K9SssV5fP9w/RuV4HrhQufy+X5SE7iPefcc998Gl99eTkXEowxRgAAABYl2h4AAACAQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACsGz1YFy4vL9fOnTvl9XqVm5urN954Q/PmzQv7vO7ubn3xxRdKSUlRQkLCYI0HoA/GGF29elWZmZlKTBy6f7f0NzcksgOwbcC5YQbBvn37THJysvnjH/9oPv30U/PMM8+YtLQ009bWFva5LS0tRhIbG1sMbC0tLYMRET0aSG4YQ3awscXK1t/cSDAm+r9cLz8/X3PnztXvfvc7Sbf+5ZKVlaWNGzdqy5YtfT7X5/MpLS1NixYt0ujRg/YGDoA+3LhxQzU1NWpvb5fT6RyS1xxIbkhkB2DbQHMj6n9rr1+/rvr6epWVlQX3JSYmqqCgQHV1dXecHwgEFAgEgo+vXr16a7DRo5WUlBTt8QBEYKi+9RFpbkhkBxCr+psbUf/m8FdffaWbN28qIyMjZH9GRoa8Xu8d53s8HjmdzuCWlZUV7ZEAxLhIc0MiO4B4Y/1TNmVlZfL5fMGtpaXF9kgAhgGyA4gvUf+WzcSJEzVq1Ci1tbWF7G9ra5PL5brjfIfDIYfDEe0xAAwjkeaGRHYA8SbqhSQ5OVl5eXmqqanRypUrJd26Oa2mpkalpaUDvv6lbUcGfI14kL1tca/HWKNb+lqjvFcPDuEksav+2eW2R5A0+LkhSdXV1VG5znC3bNmyXo+xRrf0tUZr1qwZwkli1549e6J+zUG5FX3z5s1au3atvve972nevHnatWuXOjo69LOf/WwwXg5AHCA3gJFtUArJj3/8Y3355Zd66aWX5PV69cgjj+jQoUN33LAGAN8iN4CRbdA+rF9aWhq1t1oBjAzkBjByWf+UDQAAAIUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgXdQLybZt25SQkBCyzZo1K9ovAyCOkBsARg/GRR966CF9+OGH/32R0YPyMgDiCLkBjGyD8jd+9OjRcrlcg3FpAHGK3ABGtkG5h+Szzz5TZmam7rvvPv30pz/VpUuXBuNlAMQRcgMY2aL+Dkl+fr4qKys1c+ZMtba2avv27fr+97+vs2fPKiUl5Y7zA4GAAoFA8LHf74/2SABiXKS5IZEdQLyJeiFZunRp8M9z5sxRfn6+pk2bpr/85S96+umn7zjf4/Fo+/bt0R4DwDASaW5IZAcQbwb9Y79paWn67ne/qwsXLvR4vKysTD6fL7i1tLQM9kgAYly43JDIDiDeDHohuXbtmi5evKgpU6b0eNzhcCg1NTVkAzCyhcsNiewA4k3UC8kvf/lL1dbW6t///rf+/ve/64knntCoUaP01FNPRfulAMQJcgNA1O8h+fzzz/XUU0/pypUrmjRpkh577DGdPHlSkyZNivZLAYgT5AaAqBeSffv2RfuSAOIcuQGA32UDAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrEowxJpInnDhxQjt37lR9fb1aW1tVVVWllStXBo8bY7R161b9/ve/V3t7u+bPn6+KigrNmDHjrq7v9/vldDpVWFiopKSkiL4YANHR1dWlw4cPy+fzKTU1dcDXG+zckMgOwLaB5kbE75B0dHQoNzdX5eXlPR7fsWOHXn/9db311ls6deqUxo0bp8LCQnV2dkY8HID4QG4ACGd0pE9YunSpli5d2uMxY4x27dqlF154QStWrJAk7dmzRxkZGdq/f79Wr149sGkBDEvkBoBwonoPSXNzs7xerwoKCoL7nE6n8vPzVVdX1+NzAoGA/H5/yAZg5OhPbkhkBxBvolpIvF6vJCkjIyNkf0ZGRvDY7Twej5xOZ3DLysqK5kgAYlx/ckMiO4B4Y/1TNmVlZfL5fMGtpaXF9kgAhgGyA4gvUS0kLpdLktTW1hayv62tLXjsdg6HQ6mpqSEbgJGjP7khkR1AvIn4pta+5OTkyOVyqaamRo888oikWx/FO3XqlIqLi6PyGpe2HYnKdYa77G2Lez120H1mCCeJXcvr/q/XY9XV1UM4SexatmyZ7RGGJDck/pt/q6//5mTHLX1lB/8PuqWv/wf1V8SF5Nq1a7pw4ULwcXNzsxoaGpSenq7s7Gxt2rRJv/nNbzRjxgzl5OToxRdfVGZmZsjPHAAwspAbAMKJuJCcPn1ajz/+ePDx5s2bJUlr165VZWWlnnvuOXV0dGj9+vVqb2/XY489pkOHDmnMmDHRmxrAsEJuAAgn4kKycOFC9fXDXRMSEvTyyy/r5ZdfHtBgAOIHuQEgHOufsgEAAKCQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArIu4kJw4cULLly9XZmamEhIStH///pDj69atU0JCQsi2ZMmSaM0LYBgiNwCEE3Eh6ejoUG5ursrLy3s9Z8mSJWptbQ1u77zzzoCGBDC8kRsAwhkd6ROWLl2qpUuX9nmOw+GQy+Xq91AA4gu5ASCcQbmH5Pjx45o8ebJmzpyp4uJiXblyZTBeBkAcITeAkS3id0jCWbJkiZ588knl5OTo4sWLev7557V06VLV1dVp1KhRd5wfCAQUCASCj/1+f7RHAhDjIs0NiewA4k3UC8nq1auDf3744Yc1Z84c3X///Tp+/LgWLVp0x/kej0fbt2+P9hgAhpFIc0MiO4B4M+gf+73vvvs0ceJEXbhwocfjZWVl8vl8wa2lpWWwRwIQ48LlhkR2APEm6u+Q3O7zzz/XlStXNGXKlB6POxwOORyOwR4DwDASLjcksgOINxEXkmvXroX8q6W5uVkNDQ1KT09Xenq6tm/frqKiIrlcLl28eFHPPfecpk+frsLCwqgODmD4IDcAhBNxITl9+rQef/zx4OPNmzdLktauXauKigo1Njbq7bffVnt7uzIzM7V48WL9+te/5l8ywAhGbgAIJ+JCsnDhQhljej1++PDhAQ0EIP6QGwDC4XfZAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6yIqJB6PR3PnzlVKSoomT56slStXqqmpKeSczs5OlZSUaMKECRo/fryKiorU1tYW1aEBDC9kB4BwEowx5m5PXrJkiVavXq25c+fqxo0bev7553X27FmdO3dO48aNkyQVFxfrvffeU2VlpZxOp0pLS5WYmKiPP/74rl7D7/fL6XSqsLBQSUlJ/fuqAAxIV1eXDh8+LJ/Pp9TU1AFfj+wA4t9AcyOiQnK7L7/8UpMnT1Ztba0WLFggn8+nSZMmae/evVq1apUk6fz583rggQdUV1enRx99NOw1CRXAvmgXktuRHUD8GWhuDOgeEp/PJ0lKT0+XJNXX16urq0sFBQXBc2bNmqXs7GzV1dUN5KUAxBGyA8DtRvf3id3d3dq0aZPmz5+v2bNnS5K8Xq+Sk5OVlpYWcm5GRoa8Xm+P1wkEAgoEAsHHfr+/vyMBGAbIDgA96fc7JCUlJTp79qz27ds3oAE8Ho+cTmdwy8rKGtD1AMQ2sgNAT/pVSEpLS1VdXa1jx45p6tSpwf0ul0vXr19Xe3t7yPltbW1yuVw9XqusrEw+ny+4tbS09GckAMMA2QGgNxEVEmOMSktLVVVVpaNHjyonJyfkeF5enpKSklRTUxPc19TUpEuXLsntdvd4TYfDodTU1JANQHwhOwCEE9E9JCUlJdq7d68OHDiglJSU4Pd2nU6nxo4dK6fTqaefflqbN29Wenq6UlNTtXHjRrnd7ru6S/5urFmzJirXGe727NnT67Hq6uohnCR2LVu2rNdjrNEtfa1RNMVCduS9ejAq1xnu6p9d3uuxS9uODOEksSt72+Jej7FGt/S1Rv0VUSGpqKiQJC1cuDBk/+7du7Vu3TpJ0quvvqrExEQVFRUpEAiosLBQb775ZlSGBTA8kR0AwomokNzNjywZM2aMysvLVV5e3u+hAMQXsgNAOPwuGwAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWBdRIfF4PJo7d65SUlI0efJkrVy5Uk1NTSHnLFy4UAkJCSHbhg0bojo0gOGF7AAQTkSFpLa2ViUlJTp58qQ++OADdXV1afHixero6Ag575lnnlFra2tw27FjR1SHBjC8kB0AwhkdycmHDh0KeVxZWanJkyervr5eCxYsCO6/55575HK5ojMhgGGP7AAQzoDuIfH5fJKk9PT0kP1/+tOfNHHiRM2ePVtlZWX6+uuve71GIBCQ3+8P2QDEN7IDwO0ieofkf3V3d2vTpk2aP3++Zs+eHdz/k5/8RNOmTVNmZqYaGxv1q1/9Sk1NTfrb3/7W43U8Ho+2b9/e3zEADDNkB4Ce9LuQlJSU6OzZs/roo49C9q9fvz7454cfflhTpkzRokWLdPHiRd1///13XKesrEybN28OPvb7/crKyurvWABiHNkBoCf9KiSlpaWqrq7WiRMnNHXq1D7Pzc/PlyRduHChx1BxOBxyOBz9GQPAMEN2AOhNRIXEGKONGzeqqqpKx48fV05OTtjnNDQ0SJKmTJnSrwEBDH9kB4BwIiokJSUl2rt3rw4cOKCUlBR5vV5JktPp1NixY3Xx4kXt3btXP/rRjzRhwgQ1Njbq2Wef1YIFCzRnzpxB+QIAxD6yA0A4ERWSiooKSbd+gNH/2r17t9atW6fk5GR9+OGH2rVrlzo6OpSVlaWioiK98MILURsYwPBDdgAIJ+Jv2fQlKytLtbW1AxoIQPwhOwCEw++yAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1kVUSCoqKjRnzhylpqYqNTVVbrdb77//fvB4Z2enSkpKNGHCBI0fP15FRUVqa2uL+tAAhheyA0A4CcYYc7cnHzx4UKNGjdKMGTNkjNHbb7+tnTt36syZM3rooYdUXFys9957T5WVlXI6nSotLVViYqI+/vjjux7I7/fL6XSqsLBQSUlJ/fqiAAxMV1eXDh8+LJ/Pp9TU1AFfj+wA4t9AcyOiQtKT9PR07dy5U6tWrdKkSZO0d+9erVq1SpJ0/vx5PfDAA6qrq9Ojjz56V9cjVAD7ol1IekJ2APFloLnR73tIbt68qX379qmjo0Nut1v19fXq6upSQUFB8JxZs2YpOztbdXV1/X0ZAHGG7ADQk9GRPuGTTz6R2+1WZ2enxo8fr6qqKj344INqaGhQcnKy0tLSQs7PyMiQ1+vt9XqBQECBQCD42O/3RzoSgGGA7ADQl4jfIZk5c6YaGhp06tQpFRcXa+3atTp37ly/B/B4PHI6ncEtKyur39cCELvIDgB9ibiQJCcna/r06crLy5PH41Fubq5ee+01uVwuXb9+Xe3t7SHnt7W1yeVy9Xq9srIy+Xy+4NbS0hLxFwEg9pEdAPoS8bdsbtfd3a1AIKC8vDwlJSWppqZGRUVFkqSmpiZdunRJbre71+c7HA45HI7g42/vsb1x48ZARwPQT9/+/RvgPe99IjuA+DLg3DAR2LJli6mtrTXNzc2msbHRbNmyxSQkJJgjR44YY4zZsGGDyc7ONkePHjWnT582brfbuN3uSF7CtLS0GElsbGwxsLW0tET095fsYGNj629uRPQOyeXLl7VmzRq1trbK6XRqzpw5Onz4sH74wx9Kkl599VUlJiaqqKhIgUBAhYWFevPNNyN5CWVmZqqlpUUpKSlKSEiQ3+9XVlaWWlpaBu3jh8Mda9Q31ie829fIGKOrV68qMzMzKtcnO2IP6xMeaxTe/65RSkrKgHJjwD+HZLB9+7MFBvPnIQx3rFHfWJ/w4nGN4vFriibWJzzWKLxorhG/ywYAAFhHIQEAANbFfCFxOBzaunVryN30CMUa9Y31CS8e1ygev6ZoYn3CY43Ci+Yaxfw9JAAAIP7F/DskAAAg/lFIAACAdRQSAABgHYUEAABYF9OFpLy8XN/5znc0ZswY5efn6x//+Iftkaw5ceKEli9frszMTCUkJGj//v0hx40xeumllzRlyhSNHTtWBQUF+uyzz+wMa4nH49HcuXOVkpKiyZMna+XKlWpqago5p7OzUyUlJZowYYLGjx+voqIitbW1WZp4aFVUVGjOnDlKTU1Vamqq3G633n///eDxeFobsuO/yI6+kRvhDVV2xGwh+fOf/6zNmzdr69at+uc//6nc3FwVFhbq8uXLtkezoqOjQ7m5uSovL+/x+I4dO/T666/rrbfe0qlTpzRu3DgVFhaqs7NziCe1p7a2ViUlJTp58qQ++OADdXV1afHixero6Aie8+yzz+rgwYN69913VVtbqy+++EJPPvmkxamHztSpU/XKK6+ovr5ep0+f1g9+8AOtWLFCn376qaT4WRuyIxTZ0TdyI7why45+/QacITBv3jxTUlISfHzz5k2TmZlpPB6PxaligyRTVVUVfNzd3W1cLpfZuXNncF97e7txOBzmnXfesTBhbLh8+bKRZGpra40xt9YkKSnJvPvuu8Fz/vWvfxlJpq6uztaYVt17773mD3/4Q1ytDdnRO7IjPHLj7gxGdsTkOyTXr19XfX29CgoKgvsSExNVUFCguro6i5PFpubmZnm93pD1cjqdys/PH9Hr5fP5JEnp6emSpPr6enV1dYWs06xZs5SdnT3i1unmzZvat2+fOjo65Ha742ZtyI7IkB13Ijf6NpjZEdFv+x0qX331lW7evKmMjIyQ/RkZGTp//rylqWKX1+uVpB7X69tjI013d7c2bdqk+fPna/bs2ZJurVNycrLS0tJCzh1J6/TJJ5/I7Xars7NT48ePV1VVlR588EE1NDTExdqQHZEhO0KRG70biuyIyUICDFRJSYnOnj2rjz76yPYoMWXmzJlqaGiQz+fTX//6V61du1a1tbW2xwJiArnRu6HIjpj8ls3EiRM1atSoO+7SbWtrk8vlsjRV7Pp2TVivW0pLS1VdXa1jx45p6tSpwf0ul0vXr19Xe3t7yPkjaZ2Sk5M1ffp05eXlyePxKDc3V6+99lrcrA3ZERmy47/Ijb4NRXbEZCFJTk5WXl6eampqgvu6u7tVU1Mjt9ttcbLYlJOTI5fLFbJefr9fp06dGlHrZYxRaWmpqqqqdPToUeXk5IQcz8vLU1JSUsg6NTU16dKlSyNqnf5Xd3e3AoFA3KwN2REZsoPc6K9ByY7o3ncbPfv27TMOh8NUVlaac+fOmfXr15u0tDTj9Xptj2bF1atXzZkzZ8yZM2eMJPPb3/7WnDlzxvznP/8xxhjzyiuvmLS0NHPgwAHT2NhoVqxYYXJycsw333xjefKhU1xcbJxOpzl+/LhpbW0Nbl9//XXwnA0bNpjs7Gxz9OhRc/r0aeN2u43b7bY49dDZsmWLqa2tNc3NzaaxsdFs2bLFJCQkmCNHjhhj4mdtyI5QZEffyI3whio7YraQGGPMG2+8YbKzs01ycrKZN2+eOXnypO2RrDl27JiRdMe2du1aY8ytj++9+OKLJiMjwzgcDrNo0SLT1NRkd+gh1tP6SDK7d+8OnvPNN9+YX/ziF+bee+8199xzj3niiSdMa2urvaGH0M9//nMzbdo0k5ycbCZNmmQWLVoUDBRj4mttyI7/Ijv6Rm6EN1TZkWCMMf18xwYAACAqYvIeEgAAMLJQSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFj3/zMn9TxTSfy6AAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiQAAAEPCAYAAABycN8YAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAZdUlEQVR4nO3dcUxV9/3/8RcoXK3CpahwJYKj1WlbC02Y0hs7YycTWWy0xWR2S8StqZGBieW7dNK0dXZLbqP5rbYdpcmWSU1m2boMTWmqbVEx7dBFJqHWSaphk6aArQkXpeWK8vn9YXq7q8D1woXP5fJ8JCfxnnPuuW8+ja++vJwLMcYYIwAAAItibQ8AAABAIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWTR6tC1dUVGjXrl3q6OhQdna2Xn31VS1ZsiTo8/r7+/X5558rISFBMTExozUegCEYY3T58mWlpaUpNnbs/t0y3NyQyA7AthHnhhkF1dXVJj4+3vzpT38yn3zyiXnyySdNUlKS6ezsDPrctrY2I4mNjS0Ctra2ttGIiAGNJDeMITvY2CJlG25uxBgT/l+ul5ubq8WLF+v3v/+9pBv/cklPT9eWLVu0bdu2IZ/r9XqVlJSkFStWaPLkUXsDB8AQrl27prq6OnV1dcnpdI7Ja44kNySyA7BtpLkR9r+1V69eVWNjo8rLy/37YmNjlZeXp4aGhlvO9/l88vl8/seXL1++MdjkyYqLiwv3eABCMFbf+gg1NySyA4hUw82NsH9z+Msvv9T169eVmpoasD81NVUdHR23nO/xeOR0Ov1benp6uEcCEOFCzQ2J7ACijfVP2ZSXl8vr9fq3trY22yMBGAfIDiC6hP1bNjNnztSkSZPU2dkZsL+zs1Mul+uW8x0OhxwOR7jHADCOhJobEtkBRJuwF5L4+Hjl5OSorq5Oa9eulXTj5rS6ujqVlpaO+PobNmwY8TWiwd69ewc9xhrdMNQavd1zcQwniVyPTEuxPYKk0c8NSaqtrQ3Ldca71atXD3qMNbqBNQpuqDUarlG5Fb2srExFRUX63ve+pyVLlmj37t3q6enRz372s9F4OQBRgNwAJrZRKSQ//vGP9cUXX+j5559XR0eHHnjgAR08ePCWG9YA4BvkBjCxjdqH9UtLS8P2ViuAiYHcACYu65+yAQAAoJAAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACsC3sh+fWvf62YmJiAbeHCheF+GQBRhNwAMHk0Lnrffffpgw8++PZFJo/KywCIIuQGMLGNyt/4yZMny+VyjcalAUQpcgOY2EblHpJPP/1UaWlpuuuuu/TTn/5UFy5cGI2XARBFyA1gYgv7OyS5ubmqqqrSggUL1N7erh07duj73/++Tp8+rYSEhFvO9/l88vl8/sfd3d3hHglAhAs1NySyA4g2YS8kBQUF/j9nZWUpNzdXc+fO1V//+lc98cQTt5zv8Xi0Y8eOcI8BYBwJNTcksgOINqP+sd+kpCR997vf1blz5wY8Xl5eLq/X69/a2tpGeyQAES5YbkhkBxBtRr2QXLlyRefPn9fs2bMHPO5wOJSYmBiwAZjYguWGRHYA0SbsheSXv/yl6uvr9Z///Ef/+Mc/9Oijj2rSpEl6/PHHw/1SAKIEuQEg7PeQfPbZZ3r88cd16dIlzZo1Sw899JCOHz+uWbNmhfulAEQJcgNA2AtJdXV1uC8JIMqRGwD4XTYAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALAuxhhjQnnCsWPHtGvXLjU2Nqq9vV01NTVau3at/7gxRtu3b9cf/vAHdXV1aenSpaqsrNT8+fNv6/rd3d1yOp3Kz89XXFxcSF8MgPDo6+vToUOH5PV6lZiYOOLrjXZuSGQHYNtIcyPkd0h6enqUnZ2tioqKAY/v3LlTr7zyil5//XWdOHFC06ZNU35+vnp7e0MeDkB0IDcABDM51CcUFBSooKBgwGPGGO3evVvPPvus1qxZI0nau3evUlNTtX//fq1fv35k0wIYl8gNAMGE9R6S1tZWdXR0KC8vz7/P6XQqNzdXDQ0NAz7H5/Opu7s7YAMwcQwnNySyA4g2YS0kHR0dkqTU1NSA/ampqf5jN/N4PHI6nf4tPT09nCMBiHDDyQ2J7ACijfVP2ZSXl8vr9fq3trY22yMBGAfIDiC6hLWQuFwuSVJnZ2fA/s7OTv+xmzkcDiUmJgZsACaO4eSGRHYA0Sbkm1qHkpmZKZfLpbq6Oj3wwAOSbnwU78SJEyouLg7La9Qu/H9huc54t/rs/w16rLa2dgwniVyrV68e9BhrdMNQazRWxiI3JP6bf4O/F8ENuUb8P0jS0P8PGq6QC8mVK1d07tw5/+PW1lY1NTUpOTlZGRkZ2rp1q377299q/vz5yszM1HPPPae0tLSAnzkAYGIhNwAEE3IhOXnypB5++GH/47KyMklSUVGRqqqq9PTTT6unp0ebNm1SV1eXHnroIR08eFBTpkwJ39QAxhVyA0AwIReS5cuXa6gf7hoTE6MXXnhBL7zwwogGAxA9yA0AwVj/lA0AAACFBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYF3IheTYsWN65JFHlJaWppiYGO3fvz/g+MaNGxUTExOwrVq1KlzzAhiHyA0AwYRcSHp6epSdna2KiopBz1m1apXa29v925tvvjmiIQGMb+QGgGAmh/qEgoICFRQUDHmOw+GQy+Ua9lAAogu5ASCYUbmH5OjRo0pJSdGCBQtUXFysS5cujcbLAIgi5AYwsYX8Dkkwq1at0mOPPabMzEydP39ezzzzjAoKCtTQ0KBJkybdcr7P55PP5/M/7u7uDvdIACJcqLkhkR1AtAl7IVm/fr3/z/fff7+ysrJ099136+jRo1qxYsUt53s8Hu3YsSPcYwAYR0LNDYnsAKLNqH/s96677tLMmTN17ty5AY+Xl5fL6/X6t7a2ttEeCUCEC5YbEtkBRJuwv0Nys88++0yXLl3S7NmzBzzucDjkcDhGewwA40iw3JDIDiDahFxIrly5EvCvltbWVjU1NSk5OVnJycnasWOHCgsL5XK5dP78eT399NOaN2+e8vPzwzo4gPGD3AAQTMiF5OTJk3r44Yf9j8vKyiRJRUVFqqysVHNzs9544w11dXUpLS1NK1eu1G9+8xv+JQNMYOQGgGBCLiTLly+XMWbQ44cOHRrRQACiD7kBIBh+lw0AALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALAupELi8Xi0ePFiJSQkKCUlRWvXrlVLS0vAOb29vSopKdGMGTM0ffp0FRYWqrOzM6xDAxhfyA4AwcQYY8ztnrxq1SqtX79eixcv1rVr1/TMM8/o9OnTOnPmjKZNmyZJKi4u1jvvvKOqqio5nU6VlpYqNjZWH3300W29Rnd3t5xOp/Lz8xUXFze8rwrAiPT19enQoUPyer1KTEwc8fXIDiD6jTQ3QiokN/viiy+UkpKi+vp6LVu2TF6vV7NmzdK+ffu0bt06SdLZs2d1zz33qKGhQQ8++GDQaxIqgH3hLiQ3IzuA6DPS3BjRPSRer1eSlJycLElqbGxUX1+f8vLy/OcsXLhQGRkZamhoGMlLAYgiZAeAm00e7hP7+/u1detWLV26VIsWLZIkdXR0KD4+XklJSQHnpqamqqOjY8Dr+Hw++Xw+/+Pu7u7hjgRgHCA7AAxk2O+QlJSU6PTp06qurh7RAB6PR06n07+lp6eP6HoAIhvZAWAgwyokpaWlqq2t1ZEjRzRnzhz/fpfLpatXr6qrqyvg/M7OTrlcrgGvVV5eLq/X69/a2tqGMxKAcYDsADCYkAqJMUalpaWqqanR4cOHlZmZGXA8JydHcXFxqqur8+9raWnRhQsX5Ha7B7ymw+FQYmJiwAYgupAdAIIJ6R6SkpIS7du3TwcOHFBCQoL/e7tOp1NTp06V0+nUE088obKyMiUnJysxMVFbtmyR2+2+rbvkb0dtbW1YrjPerV69etBjrNENrFFwQ61ROEVCdrzdczEs1xnvHpmWMuixDRs2jOEkkWvv3r2DHmONbhhqjYYrpEJSWVkpSVq+fHnA/j179mjjxo2SpJdeekmxsbEqLCyUz+dTfn6+XnvttbAMC2B8IjsABBNSIbmdH1kyZcoUVVRUqKKiYthDAYguZAeAYPhdNgAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsC6kQuLxeLR48WIlJCQoJSVFa9euVUtLS8A5y5cvV0xMTMC2efPmsA4NYHwhOwAEE1Ihqa+vV0lJiY4fP673339ffX19WrlypXp6egLOe/LJJ9Xe3u7fdu7cGdahAYwvZAeAYCaHcvLBgwcDHldVVSklJUWNjY1atmyZf/8dd9whl8sVngkBjHtkB4BgRnQPidfrlSQlJycH7P/zn/+smTNnatGiRSovL9dXX3016DV8Pp+6u7sDNgDRjewAcLOQ3iH5X/39/dq6dauWLl2qRYsW+ff/5Cc/0dy5c5WWlqbm5mb96le/UktLi/7+978PeB2Px6MdO3YMdwwA4wzZAWAgwy4kJSUlOn36tD788MOA/Zs2bfL/+f7779fs2bO1YsUKnT9/Xnffffct1ykvL1dZWZn/cXd3t9LT04c7FoAIR3YAGMiwCklpaalqa2t17NgxzZkzZ8hzc3NzJUnnzp0bMFQcDoccDsdwxgAwzpAdAAYTUiExxmjLli2qqanR0aNHlZmZGfQ5TU1NkqTZs2cPa0AA4x/ZASCYkApJSUmJ9u3bpwMHDighIUEdHR2SJKfTqalTp+r8+fPat2+ffvSjH2nGjBlqbm7WU089pWXLlikrK2tUvgAAkY/sABBMSIWksrJS0o0fYPS/9uzZo40bNyo+Pl4ffPCBdu/erZ6eHqWnp6uwsFDPPvts2AYGMP6QHQCCCflbNkNJT09XfX39iAYCEH3IDgDB8LtsAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdSEVksrKSmVlZSkxMVGJiYlyu9169913/cd7e3tVUlKiGTNmaPr06SosLFRnZ2fYhwYwvpAdAIKJMcaY2z357bff1qRJkzR//nwZY/TGG29o165dOnXqlO677z4VFxfrnXfeUVVVlZxOp0pLSxUbG6uPPvrotgfq7u6W0+lUfn6+4uLihvVFARiZvr4+HTp0SF6vV4mJiSO+HtkBRL+R5kZIhWQgycnJ2rVrl9atW6dZs2Zp3759WrdunSTp7Nmzuueee9TQ0KAHH3zwtq5HqAD2hbuQDITsAKLLSHNj2PeQXL9+XdXV1erp6ZHb7VZjY6P6+vqUl5fnP2fhwoXKyMhQQ0PDcF8GQJQhOwAMZHKoT/j444/ldrvV29ur6dOnq6amRvfee6+ampoUHx+vpKSkgPNTU1PV0dEx6PV8Pp98Pp//cXd3d6gjARgHyA4AQwn5HZIFCxaoqalJJ06cUHFxsYqKinTmzJlhD+DxeOR0Ov1benr6sK8FIHKRHQCGEnIhiY+P17x585STkyOPx6Ps7Gy9/PLLcrlcunr1qrq6ugLO7+zslMvlGvR65eXl8nq9/q2trS3kLwJA5CM7AAwl5G/Z3Ky/v18+n085OTmKi4tTXV2dCgsLJUktLS26cOGC3G73oM93OBxyOBz+x9/cY3vt2rWRjgZgmL75+zfCe96HRHYA0WXEuWFCsG3bNlNfX29aW1tNc3Oz2bZtm4mJiTHvvfeeMcaYzZs3m4yMDHP48GFz8uRJ43a7jdvtDuUlTFtbm5HExsYWAVtbW1tIf3/JDjY2tuHmRkjvkFy8eFEbNmxQe3u7nE6nsrKydOjQIf3whz+UJL300kuKjY1VYWGhfD6f8vPz9dprr4XyEkpLS1NbW5sSEhIUExOj7u5upaenq62tbdQ+fjjesUZDY32Cu3mNjDG6fPmy0tLSwnJ9siPysD7BsUbB/e8aJSQkjCg3RvxzSEbbNz9bYDR/HsJ4xxoNjfUJLhrXKBq/pnBifYJjjYIL5xrxu2wAAIB1FBIAAGBdxBcSh8Oh7du3B9xNj0Cs0dBYn+CicY2i8WsKJ9YnONYouHCuUcTfQwIAAKJfxL9DAgAAoh+FBAAAWEchAQAA1lFIAACAdRFdSCoqKvSd73xHU6ZMUW5urv75z3/aHsmaY8eO6ZFHHlFaWppiYmK0f//+gOPGGD3//POaPXu2pk6dqry8PH366ad2hrXE4/Fo8eLFSkhIUEpKitauXauWlpaAc3p7e1VSUqIZM2Zo+vTpKiwsVGdnp6WJx1ZlZaWysrKUmJioxMREud1uvfvuu/7j0bQ2ZMe3yI6hkRvBjVV2RGwh+ctf/qKysjJt375d//rXv5Sdna38/HxdvHjR9mhW9PT0KDs7WxUVFQMe37lzp1555RW9/vrrOnHihKZNm6b8/Hz19vaO8aT21NfXq6SkRMePH9f777+vvr4+rVy5Uj09Pf5znnrqKb399tt66623VF9fr88//1yPPfaYxanHzpw5c/Tiiy+qsbFRJ0+e1A9+8AOtWbNGn3zyiaToWRuyIxDZMTRyI7gxy45h/QacMbBkyRJTUlLif3z9+nWTlpZmPB6PxakigyRTU1Pjf9zf329cLpfZtWuXf19XV5dxOBzmzTfftDBhZLh48aKRZOrr640xN9YkLi7OvPXWW/5z/v3vfxtJpqGhwdaYVt15553mj3/8Y1StDdkxOLIjOHLj9oxGdkTkOyRXr15VY2Oj8vLy/PtiY2OVl5enhoYGi5NFptbWVnV0dASsl9PpVG5u7oReL6/XK0lKTk6WJDU2Nqqvry9gnRYuXKiMjIwJt07Xr19XdXW1enp65Ha7o2ZtyI7QkB23IjeGNprZEdJv+x0rX375pa5fv67U1NSA/ampqTp79qylqSJXR0eHJA24Xt8cm2j6+/u1detWLV26VIsWLZJ0Y53i4+OVlJQUcO5EWqePP/5Ybrdbvb29mj59umpqanTvvfeqqakpKtaG7AgN2RGI3BjcWGRHRBYSYKRKSkp0+vRpffjhh7ZHiSgLFixQU1OTvF6v/va3v6moqEj19fW2xwIiArkxuLHIjoj8ls3MmTM1adKkW+7S7ezslMvlsjRV5PpmTVivG0pLS1VbW6sjR45ozpw5/v0ul0tXr15VV1dXwPkTaZ3i4+M1b9485eTkyOPxKDs7Wy+//HLUrA3ZERqy41vkxtDGIjsispDEx8crJydHdXV1/n39/f2qq6uT2+22OFlkyszMlMvlCliv7u5unThxYkKtlzFGpaWlqqmp0eHDh5WZmRlwPCcnR3FxcQHr1NLSogsXLkyodfpf/f398vl8UbM2ZEdoyA5yY7hGJTvCe99t+FRXVxuHw2GqqqrMmTNnzKZNm0xSUpLp6OiwPZoVly9fNqdOnTKnTp0ykszvfvc7c+rUKfPf//7XGGPMiy++aJKSksyBAwdMc3OzWbNmjcnMzDRff/215cnHTnFxsXE6nebo0aOmvb3dv3311Vf+czZv3mwyMjLM4cOHzcmTJ43b7TZut9vi1GNn27Ztpr6+3rS2tprm5mazbds2ExMTY9577z1jTPSsDdkRiOwYGrkR3FhlR8QWEmOMefXVV01GRoaJj483S5YsMcePH7c9kjVHjhwxkm7ZioqKjDE3Pr733HPPmdTUVONwOMyKFStMS0uL3aHH2EDrI8ns2bPHf87XX39tfvGLX5g777zT3HHHHebRRx817e3t9oYeQz//+c/N3LlzTXx8vJk1a5ZZsWKFP1CMia61ITu+RXYMjdwIbqyyI8YYY4b5jg0AAEBYROQ9JAAAYGKhkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALDu/wPBqPU8AVQmZgAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TEST:\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiQAAAEPCAYAAABycN8YAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAZnElEQVR4nO3dcUzU9/3H8RconFbhKCqcRHC0Om1roQlTerMz9icTWTTaYjK7LerW1MjAxJKlk6attVtyjSatbUdpsmVSs1m3LkOjTbUtKqYtusgk1jpJNWzSFLA14U5pOVE+vz9MbzsFzoODz3E8H8k38b7f733vzafx1ZfH9yDOGGMEAABgUbztAQAAACgkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwLqxQ3XhyspKbdu2TW1tbcrNzdVrr72mefPmhXxeT0+PvvjiCyUlJSkuLm6oxgPQD2OMLl++rIyMDMXHD9+/WwaaGxLZAdg26NwwQ2D37t0mMTHR/PGPfzSffvqpeeKJJ0xKSoppb28P+dyWlhYjiY2NLQq2lpaWoYiIXg0mN4whO9jYomUbaG7EGRP5X66Xn5+vuXPn6ne/+52kG/9yyczM1IYNG7Rp06Z+n+v1epWSkqJFixZp7NghewMHQD+uXbum2tpadXR0yOl0DstrDiY3JLIDsG2wuRHxv7VXr15VQ0ODKioqAvvi4+NVUFCg+vr6W873+/3y+/2Bx5cvX74x2NixSkhIiPR4AMIwXN/6CDc3JLIDiFYDzY2If3P4q6++0vXr15Wenh60Pz09XW1tbbec7/F45HQ6A1tmZmakRwIQ5cLNDYnsAGKN9U/ZVFRUyOv1BraWlhbbIwEYAcgOILZE/Fs2kydP1pgxY9Te3h60v729XS6X65bzHQ6HHA5HpMcAMIKEmxsS2QHEmogXksTEROXl5am2tlYrVqyQdOPmtNraWpWVlQ36+n/6uHzQ14gFP/v+S30eu/D8e8M4SfTKen5xn8dWr149jJNEr507d9oeQdLQ54Yk7d+/PyLXGemWLl3a5zHW6Ib+1mjfdtZIkpZt7HuNBmpIbkUvLy/XmjVr9L3vfU/z5s3T9u3b1dnZqZ///OdD8XIAYgC5AYxuQ1JIfvzjH+vLL7/Uc889p7a2Nj3wwAM6cODALTesAcC3yA1gdBuyD+uXlZVF7K1WAKMDuQGMXtY/ZQMAAEAhAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWBfxQvL8888rLi4uaJs9e3akXwZADCE3AIwdioved999+uCDD/77ImOH5GUAxBByAxjdhuRv/NixY+VyuYbi0gBiFLkBjG5Dcg/JZ599poyMDN1111366U9/qgsXLgzFywCIIeQGMLpF/B2S/Px8VVdXa9asWWptbdWWLVv0gx/8QKdPn1ZSUtIt5/v9fvn9/sBjn88X6ZEARLlwc0MiO4BYE/FCUlRUFPhzTk6O8vPzNX36dP31r3/V448/fsv5Ho9HW7ZsifQYAEaQcHNDIjuAWDPkH/tNSUnRd7/7XZ07d67X4xUVFfJ6vYGtpaVlqEcCEOVC5YZEdgCxZsgLyZUrV3T+/HlNnTq11+MOh0PJyclBG4DRLVRuSGQHEGsiXkh+9atfqa6uTv/+97/18ccf65FHHtGYMWP02GOPRfqlAMQIcgNAxO8h+fzzz/XYY4/p0qVLmjJlih566CEdO3ZMU6ZMifRLAYgR5AaAiBeS3bt3R/qSAGIcuQGA32UDAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADr4owxJpwnHD16VNu2bVNDQ4NaW1tVU1OjFStWBI4bY7R582b9/ve/V0dHh+bPn6+qqirNnDnztq7v8/nkdDpVWFiohISEsL4YAJHR3d2tgwcPyuv1Kjk5edDXG+rckMgOwLbB5kbY75B0dnYqNzdXlZWVvR7funWrXn31Vb3xxhs6fvy4JkyYoMLCQnV1dYU9HIDYQG4ACGVsuE8oKipSUVFRr8eMMdq+fbueeeYZLV++XJK0c+dOpaena8+ePVq1atXgpgUwIpEbAEKJ6D0kzc3NamtrU0FBQWCf0+lUfn6+6uvre32O3++Xz+cL2gCMHgPJDYnsAGJNRAtJW1ubJCk9PT1of3p6euDYzTwej5xOZ2DLzMyM5EgAotxAckMiO4BYY/1TNhUVFfJ6vYGtpaXF9kgARgCyA4gtES0kLpdLktTe3h60v729PXDsZg6HQ8nJyUEbgNFjILkhkR1ArAn7ptb+ZGdny+Vyqba2Vg888ICkGx/FO378uEpKSiLyGn/6uDwi1xnpfvb9l/o8lvfyvmGcJHo1PLmsz2P79+8fxkmi19KlS22PMCy5IfHf/Fv9/TcnO27oLzv4f9AN/f0/aKDCLiRXrlzRuXPnAo+bm5vV2Nio1NRUZWVlaePGjfrtb3+rmTNnKjs7W88++6wyMjKCfuYAgNGF3AAQStiF5MSJE3r44YcDj8vLb7TFNWvWqLq6Wk899ZQ6Ozu1bt06dXR06KGHHtKBAwc0bty4yE0NYEQhNwCEEnYhWbhwofr74a5xcXF64YUX9MILLwxqMACxg9wAEIr1T9kAAABQSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANaFXUiOHj2qZcuWKSMjQ3FxcdqzZ0/Q8bVr1youLi5oW7JkSaTmBTACkRsAQgm7kHR2dio3N1eVlZV9nrNkyRK1trYGtrfeemtQQwIY2cgNAKGMDfcJRUVFKioq6vcch8Mhl8s14KEAxBZyA0AoQ3IPyZEjR5SWlqZZs2appKREly5dGoqXARBDyA1gdAv7HZJQlixZokcffVTZ2dk6f/68nn76aRUVFam+vl5jxoy55Xy/3y+/3x947PP5Ij0SgCgXbm5IZAcQayJeSFatWhX48/3336+cnBzdfffdOnLkiBYtWnTL+R6PR1u2bIn0GABGkHBzQyI7gFgz5B/7veuuuzR58mSdO3eu1+MVFRXyer2BraWlZahHAhDlQuWGRHYAsSbi75Dc7PPPP9elS5c0derUXo87HA45HI6hHgPACBIqNySyA4g1YReSK1euBP2rpbm5WY2NjUpNTVVqaqq2bNmi4uJiuVwunT9/Xk899ZRmzJihwsLCiA4OYOQgNwCEEnYhOXHihB5++OHA4/LycknSmjVrVFVVpVOnTunNN99UR0eHMjIytHjxYv3mN7/hXzLAKEZuAAgl7EKycOFCGWP6PH7w4MFBDQQg9pAbAELhd9kAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrwiokHo9Hc+fOVVJSktLS0rRixQo1NTUFndPV1aXS0lJNmjRJEydOVHFxsdrb2yM6NICRhewAEEqcMcbc7slLlizRqlWrNHfuXF27dk1PP/20Tp8+rTNnzmjChAmSpJKSEr3zzjuqrq6W0+lUWVmZ4uPj9dFHH93Wa/h8PjmdThUWFiohIWFgXxWAQenu7tbBgwfl9XqVnJw86OuRHUDsG2xuhFVIbvbll18qLS1NdXV1WrBggbxer6ZMmaJdu3Zp5cqVkqSzZ8/qnnvuUX19vR588MGQ1yRUAPsiXUhuRnYAsWewuTGoe0i8Xq8kKTU1VZLU0NCg7u5uFRQUBM6ZPXu2srKyVF9fP5iXAhBDyA4ANxs70Cf29PRo48aNmj9/vubMmSNJamtrU2JiolJSUoLOTU9PV1tbW6/X8fv98vv9gcc+n2+gIwEYAcgOAL0Z8DskpaWlOn36tHbv3j2oATwej5xOZ2DLzMwc1PUARDeyA0BvBlRIysrKtH//fh0+fFjTpk0L7He5XLp69ao6OjqCzm9vb5fL5er1WhUVFfJ6vYGtpaVlICMBGAHIDgB9CauQGGNUVlammpoaHTp0SNnZ2UHH8/LylJCQoNra2sC+pqYmXbhwQW63u9drOhwOJScnB20AYgvZASCUsO4hKS0t1a5du7R3714lJSUFvrfrdDo1fvx4OZ1OPf744yovL1dqaqqSk5O1YcMGud3u27pL/nbs274/ItcZ6ZZtXNrnsf37WSNJWrqUNQqlvzWKpGjIjtWrV0fkOiPdzp07+zx24fn3hnGS6JX1/OI+j/3p4/JhnCR6/ez7L0X8mmEVkqqqKknSwoULg/bv2LFDa9eulSS9/PLLio+PV3Fxsfx+vwoLC/X6669HZFgAIxPZASCUsArJ7fzIknHjxqmyslKVlZUDHgpAbCE7AITC77IBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1YRUSj8ejuXPnKikpSWlpaVqxYoWampqCzlm4cKHi4uKCtvXr10d0aAAjC9kBIJSwCkldXZ1KS0t17Ngxvf/+++ru7tbixYvV2dkZdN4TTzyh1tbWwLZ169aIDg1gZCE7AIQyNpyTDxw4EPS4urpaaWlpamho0IIFCwL777jjDrlcrshMCGDEIzsAhDKoe0i8Xq8kKTU1NWj/n//8Z02ePFlz5sxRRUWFvv766z6v4ff75fP5gjYAsY3sAHCzsN4h+V89PT3auHGj5s+frzlz5gT2/+QnP9H06dOVkZGhU6dO6de//rWampr097//vdfreDwebdmyZaBjABhhyA4AvRlwISktLdXp06f14YcfBu1ft25d4M/333+/pk6dqkWLFun8+fO6++67b7lORUWFysvLA499Pp8yMzMHOhaAKEd2AOjNgApJWVmZ9u/fr6NHj2ratGn9npufny9JOnfuXK+h4nA45HA4BjIGgBGG7ADQl7AKiTFGGzZsUE1NjY4cOaLs7OyQz2lsbJQkTZ06dUADAhj5yA4AoYRVSEpLS7Vr1y7t3btXSUlJamtrkyQ5nU6NHz9e58+f165du/SjH/1IkyZN0qlTp/Tkk09qwYIFysnJGZIvAED0IzsAhBJWIamqqpJ04wcY/a8dO3Zo7dq1SkxM1AcffKDt27ers7NTmZmZKi4u1jPPPBOxgQGMPGQHgFDC/pZNfzIzM1VXVzeogQDEHrIDQCj8LhsAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGBdWIWkqqpKOTk5Sk5OVnJystxut959993A8a6uLpWWlmrSpEmaOHGiiouL1d7eHvGhAYwsZAeAUOKMMeZ2T963b5/GjBmjmTNnyhijN998U9u2bdPJkyd13333qaSkRO+8846qq6vldDpVVlam+Ph4ffTRR7c9kM/nk9PpVGFhoRISEgb0RQEYnO7ubh08eFBer1fJycmDvh7ZAcS+weZGWIWkN6mpqdq2bZtWrlypKVOmaNeuXVq5cqUk6ezZs7rnnntUX1+vBx988LauR6gA9kW6kPSG7ABiy2BzY8D3kFy/fl27d+9WZ2en3G63Ghoa1N3drYKCgsA5s2fPVlZWlurr6wf6MgBiDNkBoDdjw33CJ598Irfbra6uLk2cOFE1NTW699571djYqMTERKWkpASdn56erra2tj6v5/f75ff7A499Pl+4IwEYAcgOAP0J+x2SWbNmqbGxUcePH1dJSYnWrFmjM2fODHgAj8cjp9MZ2DIzMwd8LQDRi+wA0J+wC0liYqJmzJihvLw8eTwe5ebm6pVXXpHL5dLVq1fV0dERdH57e7tcLlef16uoqJDX6w1sLS0tYX8RAKIf2QGgP2F/y+ZmPT098vv9ysvLU0JCgmpra1VcXCxJampq0oULF+R2u/t8vsPhkMPhCDz+9h7ba9euDXY0AAP07d+/Qd7z3i+yA4gtg84NE4ZNmzaZuro609zcbE6dOmU2bdpk4uLizHvvvWeMMWb9+vUmKyvLHDp0yJw4ccK43W7jdrvDeQnT0tJiJLGxsUXB1tLSEtbfX7KDjY1toLkR1jskFy9e1OrVq9Xa2iqn06mcnBwdPHhQP/zhDyVJL7/8suLj41VcXCy/36/CwkK9/vrr4byEMjIy1NLSoqSkJMXFxcnn8ykzM1MtLS1D9vHDkY416h/rE9rNa2SM0eXLl5WRkRGR65Md0Yf1CY01Cu1/1ygpKWlQuTHon0My1L792QJD+fMQRjrWqH+sT2ixuEax+DVFEusTGmsUWiTXiN9lAwAArKOQAAAA66K+kDgcDm3evDnobnoEY436x/qEFotrFItfUySxPqGxRqFFco2i/h4SAAAQ+6L+HRIAABD7KCQAAMA6CgkAALCOQgIAAKyL6kJSWVmp73znOxo3bpzy8/P1j3/8w/ZI1hw9elTLli1TRkaG4uLitGfPnqDjxhg999xzmjp1qsaPH6+CggJ99tlndoa1xOPxaO7cuUpKSlJaWppWrFihpqamoHO6urpUWlqqSZMmaeLEiSouLlZ7e7uliYdXVVWVcnJylJycrOTkZLndbr377ruB47G0NmTHf5Ed/SM3Qhuu7IjaQvKXv/xF5eXl2rx5s/75z38qNzdXhYWFunjxou3RrOjs7FRubq4qKyt7Pb5161a9+uqreuONN3T8+HFNmDBBhYWF6urqGuZJ7amrq1NpaamOHTum999/X93d3Vq8eLE6OzsD5zz55JPat2+f3n77bdXV1emLL77Qo48+anHq4TNt2jS9+OKLamho0IkTJ/R///d/Wr58uT799FNJsbM2ZEcwsqN/5EZow5YdA/oNOMNg3rx5prS0NPD4+vXrJiMjw3g8HotTRQdJpqamJvC4p6fHuFwus23btsC+jo4O43A4zFtvvWVhwuhw8eJFI8nU1dUZY26sSUJCgnn77bcD5/zrX/8ykkx9fb2tMa268847zR/+8IeYWhuyo29kR2jkxu0ZiuyIyndIrl69qoaGBhUUFAT2xcfHq6CgQPX19RYni07Nzc1qa2sLWi+n06n8/PxRvV5er1eSlJqaKklqaGhQd3d30DrNnj1bWVlZo26drl+/rt27d6uzs1Nutztm1obsCA/ZcStyo39DmR1h/bbf4fLVV1/p+vXrSk9PD9qfnp6us2fPWpoqerW1tUlSr+v17bHRpqenRxs3btT8+fM1Z84cSTfWKTExUSkpKUHnjqZ1+uSTT+R2u9XV1aWJEyeqpqZG9957rxobG2NibciO8JAdwciNvg1HdkRlIQEGq7S0VKdPn9aHH35oe5SoMmvWLDU2Nsrr9epvf/ub1qxZo7q6OttjAVGB3OjbcGRHVH7LZvLkyRozZswtd+m2t7fL5XJZmip6fbsmrNcNZWVl2r9/vw4fPqxp06YF9rtcLl29elUdHR1B54+mdUpMTNSMGTOUl5cnj8ej3NxcvfLKKzGzNmRHeMiO/yI3+jcc2RGVhSQxMVF5eXmqra0N7Ovp6VFtba3cbrfFyaJTdna2XC5X0Hr5fD4dP358VK2XMUZlZWWqqanRoUOHlJ2dHXQ8Ly9PCQkJQevU1NSkCxcujKp1+l89PT3y+/0xszZkR3jIDnJjoIYkOyJ7323k7N692zgcDlNdXW3OnDlj1q1bZ1JSUkxbW5vt0ay4fPmyOXnypDl58qSRZF566SVz8uRJ85///McYY8yLL75oUlJSzN69e82pU6fM8uXLTXZ2tvnmm28sTz58SkpKjNPpNEeOHDGtra2B7euvvw6cs379epOVlWUOHTpkTpw4Ydxut3G73RanHj6bNm0ydXV1prm52Zw6dcps2rTJxMXFmffee88YEztrQ3YEIzv6R26ENlzZEbWFxBhjXnvtNZOVlWUSExPNvHnzzLFjx2yPZM3hw4eNpFu2NWvWGGNufHzv2WefNenp6cbhcJhFixaZpqYmu0MPs97WR5LZsWNH4JxvvvnG/PKXvzR33nmnueOOO8wjjzxiWltb7Q09jH7xi1+Y6dOnm8TERDNlyhSzaNGiQKAYE1trQ3b8F9nRP3IjtOHKjjhjjBngOzYAAAAREZX3kAAAgNGFQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMC6/wem2PU8xJ4X7QAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Evaluate:** ARC Benchmark\n",
        "\n",
        "Evaluate on the available 800 tasks.\n",
        "\n",
        "**Note:** LLM temperature is set to 0 (deterministic), but your results might still vary depending on stability of the API."
      ],
      "metadata": {
        "id": "A6Jz5lc5bQRx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "success = {}\n",
        "for task_id in sorted_task_ids:\n",
        "  task_json, task_name = tasks_jsons[task_id], tasks_names[task_id]\n",
        "\n",
        "  # Lazy load: skip evals where we already have results.\n",
        "  if task_name in success:\n",
        "    continue\n",
        "\n",
        "  # Build context and expected output labels.\n",
        "  context = []\n",
        "  batch_prompts = []\n",
        "  batch_labels = []\n",
        "  train_xy, test_x, test_y = task_json_to_tokens(task_json, value_to_token)\n",
        "  test_num_tokens = np.max([len(x) + len(y) for x, y in zip(test_x, test_y)])\n",
        "  for sample in train_xy:\n",
        "    if len(context) + len(sample) + test_num_tokens > token_limit:  # Ensure both train and test examples can fit in the prompt.\n",
        "      break\n",
        "    context += sample\n",
        "\n",
        "  # There can be multiple test examples so put them in the same batch.\n",
        "  for x, y in zip(test_x, test_y):\n",
        "    batch_prompts.append(context + x)\n",
        "    batch_labels.append(y)\n",
        "\n",
        "  # Run LLM.\n",
        "  try:\n",
        "    stop_token = tokenizer.decode(sample_delim, skip_special_tokens=True)\n",
        "    max_tokens = int(np.max([len(y) for y in test_y])) + 10\n",
        "    batch_responses = LLM(batch_prompts, stop=stop_token, max_tokens=max_tokens, temperature=0)\n",
        "  except Exception as e:\n",
        "    print(task_name, f\"LLM failed. {e}\")\n",
        "    continue\n",
        "\n",
        "  # Check answers and save success rates.\n",
        "  success[task_name] = 0\n",
        "  for response, label in zip(batch_responses, batch_labels):\n",
        "    label_str = tokenizer.decode(label, skip_special_tokens=True)\n",
        "    is_success = label_str.strip() in response\n",
        "    success[task_name] += is_success / len(batch_labels)\n",
        "  success[task_name] = int(success[task_name] > 0.99)  # All test cases need to correct.\n",
        "\n",
        "  # Debug prints.\n",
        "  total_success = np.sum(list(success.values()))\n",
        "  print(task_name, \"Success:\", success[task_name], \"Total:\", f\"{total_success} / {len(success)}\")\n",
        "\n",
        "  # # Save results.\n",
        "  # result_file = f\"arc-{model}-alphabet-{'-'.join(map(str, alphabet))}.pkl\"\n",
        "  # with open(result_file, 'wb') as fid:\n",
        "  #   pickle.dump(success, fid, protocol=pickle.HIGHEST_PROTOCOL)"
      ],
      "metadata": {
        "id": "GuFgcjPxsxXG"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}